Self-attention in vision transformers performs perceptual grouping, not attention

Recently, a considerable number of studies in computer vision involve deep neural architectures called transformers. Visual processing these models incorporates computational that are claimed to implement attention mechanisms. Despite an increasing body work attempts understand the role mechanisms transformers, their effect is largely unknown. Here, we asked if transformers exhibit similar effects as those known human visual attention. To answer this question, revisited formulation and found despite name, computationally, perform special class relaxation labeling with similarity grouping effects. Additionally, whereas modern experimental findings reveal involves both feed-forward feedback mechanisms, purely architecture suggests cannot have same humans. quantify observations, evaluated performance family Our results suggest self-attention modules group figures stimuli based on features such color. Also, singleton detection experiment instance salient object detection, studied salience thought be utilized We generally, transformer-based assign more either distractors or ground, opposite salience. Together, our study perceptual organization feature not